🙇‍♀️ Problem Statement ​

🌟 Overview

Honeywell Tech Support Portal is designed to help our customers to search for solutions to problems they are facing with maintenance or installation of any part of their aircraft which is manufactured by Honeywell.

It became very essential during COVID that the users are able to use the portal efficiently due to limited in person support from Field Engineers .

A Few changes were done to the portal in Sep 2020 :

a. Browse By - Users will be able to browse Technical Solutions by the following:

• Aircraft
• Product
• ATA Chapter

b. Popular Technical Solutions - Users will be able to see a highlight of the top 8 most popular technical solutions

​

​

c. Technical Solutions Search Results

Robust, dynamic filtering capability for easier and exact search. Detailed information on each result card, so users do not need to click through an article to see high-level detail.

​

​

d. Download as PDF

There is an option to download a technical solution article as a PDF

Core Problem

1. Increase in the time spent in resolving technical cases from Q4 2020-Q2 2021
2. No streamlined method to calculate how many users are able to "self serve " themselves and how many still find it difficult to navigate after the above changes were made .( change in the definition of "activated users ")
3. Discrepancy in the "self serve " percentage across verticals

​

Solution

Develop a Tableau Dashboard which fetches the case creation data from multiple Databases and analyse trends to make the portal more efficient for the customers.

​

This Dashboard will show the number of cases created each month , time they spend browsing the portal , number of sessions / per user , time after which they created a ticket , date of ticket created , self serve ( activated users ) and self serve percentage .

🤔 Hypothesis

If we accurately calculate self serve % on our Tableu Dashboard , then our portal activation rate will increase upto 54% due to clear understanding of what information to focus on .

🎯 Goal

Increasing the active engagement on the portal will help the team to organise the portal in a way which increases the usability of the portal . It will in-turn help reduce manual labour and directly impact the revenue generated ( 1.5M dollars in 14 months projected due to reduction in manual effort)

​

✅ Success metric

a. Accuracy of case tracking

b. Adoption of Dashboard across customer and support team.

Worst Case : New calculations doesn't improve the activated user rate on the portal and increased manual efforts

Best Case : Reduction in manual efforts by atleast 30%

​

🧪 Experimental Design

TLDR : Effectively calculating the self serve % in a month and how the users are using the portal to streamline the portal

Scenarios :

1. User logins in with the email id , searches the portal , finds a solution to their problem, doesn't create a case
2. User logins in with the email id , doesn't search the portal , directly creates a case using webform or SFDC.
3. User logins in with the email id , searches the portal , views the videos or technical publications , or articles ,doesn't find a solution , creates a case using webform or SFDC

In the above listed scenarios, only scenarios 2& 3 are considered for the calculation of total cases created .

Flag No case creation but activity as 0( success ) and case created with activity or no activity as 1

Self Serve % Formula = (Sum of Zeros / Total cases created)*100

Variation Design

New Change :

1. User logins in with the email id , searches the portal , finds a solution to their problem, doesn't create a case - no change still stays irrelevant
2. User logins in with the email id , doesn't search the portal , directly creates a case using webform or SFDC. - if the case is not created within 48 hrs , this becomes irrelevant
3. User logins in with the email id , searches the portal , views the videos for >=2 mins or technical publications for 10mins , or articles ,doesn't find a solution , creates a case using webform or SFDC within 48 hrs - if the video is viewed for less than 2 mins or the publication is viewed for less than 10 mins, it becomes irrelevant. Articles were a small number in 2020 so they were excluded from the calculation
4. Dashboard shows a line chart of numbers of cases plotted against each month and filters for date , month , year , by internal and external users.

​

Flag No case creation but activity as 0( success ) and case created with activity or no activity (with new scenario 2 and 3) as 1

Self Serve % Formula = (Sum of Zeros / Total cases created)*100

Cohorts

Test Group 1 : 100 people

Release Dashboard to technical experts handling the cases on Airbus and Boeing only

Control Group : 3000 people

The rest of the experts were using manual methods to calculate the self serve % and total cases created for other products

💻 Implementation of A/A test

The results on the Tableau Dashboard after modified automated calculation vs manual calculation were the same in the test group .

💡 Post Experiment

Experiment Results

After using the numbers shown on the Dashboard continuously for 3 months , the technical experts were able to :

​

• Measured the accuracy of case tracking by comparing the discrepancies in case records before and after the implementation.
• Decreased in number of duplicate tickets for Airbus and Boeing by 10%
• Understood the user behaviour on the portal and organise the information on the portal according ( increase the count of videos since they were the most viewed out of all the solutions offered)
• Reduction in man- hours ( 30% seen after 14 months of successfully using the Dashboard )
• Introduced another tab ( after 3 months ) which gave a detailed view of the cases created by user . This helped the Technical engineers to dive deeper into the products which faced the problem often .

Release Decision

Scale the experiment to remaining aircrafts types.

Learnings

• The definition of activated user is important to measure the engagement on the portal .
• Understand the user behaviour to design the use cases and edge cases efficiently.
• Keep stakeholder in loop for any changes and train them to use the Dashboard efficiently.
• Focus on A/A testing while scaling it for different businesses.

​

Next steps

Understand how to scale for Safety Productivity Solutions

Timeline and milestones not defined yet .

​

​

​

​

​

​

​